IN THE CLAIMS : j J / / 

Please cancel Claims 5-7, 11-17, 24-26 and 30-36 without prejudice to or 
disclaimer of the recited subject matter. 

Please amend Claims 1, 8, 20, 27 and 39 as follows. 



S: 



^ 1*^/ 1- (Currently Amended) A simulator apg&ratus with which an operator plays a 
simulation with virtual object(s) in mixed reality sp Ace including a virtual space and real- 



space, said simulator comprising: 

a viewpoint detection unit adapted to/detect the location/posture of a viewpoint 
| of the operator; 

an inputting unit adapted to input a £eal space image corresponding to the 
location/posture of a viewpoint of the operator; 

a geometric information acquisition unit adapted to recognize geometric 
information of real object(s); 

a rec o gnition uni t adap te d t o r e cogniz e a current rela t iv e r e lationship between 



es for controlling the action of the virtual 



t h e virtual o bj e ct(s) and real object(s); 

a rule memory adapted to store ruji 

object(s); 

a computation unit adapted to determine the next action of the virtual object(s) 
by referring to said rule memory based on a relation among the location/posture of a 



viewpoint of the operator, location(s) of the virtual obiectfs) and the geometric information 



of the real object(s): in accordanc e with the ru 



es s t or e d in said rul e m e mory and in 
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cor re sp o nd e nce with the loca t i o n/p o s t ure elf t h e real o bj e et(s), and t o comput e th e 
loea t i o n/p o s t ure of t h e virtual obj e ct(s) af te r the d ete rmin e d ac t i o n; and 

a presentation unit adapted to! generate at least one image of the virtual 
object(s) on the basis of the location/posmire of the virtual object(s) after the determined 
action and the location/posture of the viewpoint position of the operator, and to represent 
the mixed reality space to the operator bjy superimposing the image(s) of the virtual 
; object(s)i^ 



2. (Previously Amended) ^he apparatus according to claim 1, wherein said 
presentation unit further comprises: 

an image-capturing unit a'dapted to capture real space images of said operator's 
view of real space images of said player's view of the real space; 



an image generation unit 



object(s) on said real space images; knd 

a video see-through typ^ 
reality images are displayed. 



adapted to generate mixed reality images representing 



the mixed reality space by superimposing or overlaying said image(s) of the virtual 



display that the operator wears wherein said mixed 



3. (Previously Amendejd) 
presentation unit further comprises 
wears wherein said virtual object 



The apparatus according to claim 1, wherein said 
an optical see-through type display that the operator 
s) are displayed. 



iinage(: 



4. (Previously Amended) The apparatus according to claim 1, further 

comprising, 

a status detector that detects a s/tatus of the operator; 

wherein said computation unit/ determines a next action of the virtual object in 
accordance with the rule stored in said rule memory and in correspondence with the 
location/posture of the real object and/or/the status of the operator, and computes a 
location/posture of the virtual obj ect after the determined action. 



5-7. (Cancelled). 



Is 



8. (Currently Amended) 
o bj e c t includes a plurality of objects 



'he apparatus according o claim 1 , wherein the real 
) include other operators who operate said simulator 



apparatus, and the plurali t y of another operators share a single mixed reality space with the 
operator . 



9. (Previously Amended) The apparatus according to claim 1, wherein the real 
object is an object which is fixed in position, and 

said geometric information acquisition unit comprises: 

a predetermined memlry for pre-storing location information and shape 
information of the real object; anc 

a reading unit that reads out the location information and shape information of 
the real object from said predetelnined memory as needed 
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10. (Previously Amended) The apparatus according to claim 1, wherein the 
real object is an object which is movable but does not deform, and 
said geometric information acquisition unit comprises: 
a predetermined memory for pre-storing shape information of the real object; 
a location/posture sensor for detecting a location/posture of the real object; and 
a setting unit that sets a region the real object is expected to occupy in the 
mixed real space in accordance with the detected location/posture of the real object. _" ~ 



11-17 (Cancelled). 



18. (Previously Amended) The apparatus according to claim 1, wherein said 
viewpoint detection unit detects a lc cation/posture of the head of the operator, and 

said apparatus further comprises a detector that detects a location/posture of a 
hand of the operator; and 

a recognition unit adap'ted to recognize a relative location of the hand of the 
operator with respect to the head a^ a command on the basis of an output from said 
detector. 



19. (Previously Amended) The apparatus according to claim 1, wherein said 
presentation unit comprises: 

an alignment unit that 
location/posture of the virtual < 



object 



iligns the location/posture of the real object to the 
after movement; 
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a generation unit that generates an image of the virtual object after alignment in 
correspondence with an occlusion relationship; and 
a head-mounted display device. 



20. (Currently Amended) An image processing method for a simulator 
apparatus with which an operator plays a game with virtual object(s) in a mixed reality 



etecting the location/posture of a viewpoint of 



space comprising: 

a viewpoint detection step for i 
the operator; 

an inputting step for inputting ja real space image corresponding to the 
location/posture of a viewpoint of the operator; 

a geometric information acquisition step for acquiring geometric information 
of real object(s); 

a r e cognition st e p for recognising a curr e nt r e lative r e la t i o nship between t he 



virtual obj e ct(s) and real object(s); 

a computation step for determining the next action of the virtual object(s) by 
referring to the rule memory based on a relation among the location/posture of a viewpoint 



of the operator, location(s) of the virtual obiect(s) and the geometric information of the real 



object(s); in accordanc e wi t h th e rules stcred in a rule memory, which s t ores rules for 



con t rolling the ac t ion of th e virtual o bjects), and in correspondence wi t h the 



l o cati o n/postur e o f th e r e al object(s), and c o mputing th e location/pos t ure of the virtual 
o bjec t (s) a ft er t he d e termined action; and 



a presentation step for generating at Aeast one image of the virtual object(s) on 
the basis of the location/posture of the virtual opject(s) after the determined action and the 
location/posture of the viewpoint position of the operator, and for representing the mixed 
reality space to the operator by superimposing 
operator's view of the real space. 



the image(s) of virtual object(s) on the 




' " " 21. (Previously Amended) "The n 
operator wears a video see-through type display, and said presentation step further 
comprising, 

image-capturing step for capturing real space images of said operator's view of 
the real space; 

image generation step for genera ing mixed reality images representing of the 



mixed reality space by superimposing or ov 



laying said image(s) of virtual object(s) on 



said real space images and for displaying said mixed reality images on the display. 



22. (Previously Amended) The method according claim 20, wherein the 
operator wears an optical see-through type c isplay and said presentation step representing 
the mixed reality space to the operator by displaying the image(s) of virtual object(s) on the 
display. 



composing, 



23. (Previously Amended) The method according to claim 20, further 



status detecting step for detecting status of the operator; 



wherein said computation step determines a next action of the virtual object in 
accordance with the rule stored in said rule memory and in correspondence with the 
location/posture of the real object and/or the status of the operator, and computing a 
location/posture of the virtual object after ihe determined action. 



24-26 (Cancelled) 




27. (Currently Amended) The method according to claim 20, wherein the real 
obiect(s) include other operators who pj crate the simulator apparatus, and wherein a 



us e d in t h e c o mputa t ion st e p includ e s a 
with the operator . 



plurali t y of the other operators share a single mixed reality space and the r eal o bj e ct(s) 



in 



28. (Previously Amended 
real object is an object which is fixed 

the geometric information 

pre-storing location 
predetermined memory; and 

reading out the location in 
from the predetermined memory as needed 



29. (Original) Themetho< 



plurality o f o pera t ors who o pera t e th e apparatus 



The method according to claim 20, wherein the 
position, and 
acquisition step includes the steps of: 
inforniation and shape information of the real object in a 



brmation and shape information of the real object 



according to claim 20, wherein the real object is an 



object which is movable but does not deform, and 



the geometric information acquisitid^ step includes the steps of: 
pre-storing shape information of th<; real object in a predetermined memory; 
detecting a location/posture of the ieal object by a location/posture sensor; and 
setting a region the real object is expected to occupy in the mixed real space in 
accordance wit h the detected location/posture of the real object. 




37. (Previously Amended) The method according to claim 20, wherein the 
viewpoint detection step includes the step of (detecting a location/posture of the head of the 
operator , and 

said method further comprises tl^e detection step of detecting a location/posture 
of a hand of the operator; and 

the step of recognizing a relative location of the hand of the operator with 
respect to the head as a command on the basis of an output in the detection step. 

38. (Original) The method according to claim 20, wherein the presentation 
step includes the steps of: 

aligning the location/posture of the real object to the location/posture of the 
virtual object after movement; and 

generating an image of the virt lal object after alignment in correspondence 
with an occlusion relationship. 



39. (Currently Amended) A storage medium which stores a program of an 
image processing method for a simulator apparatus with which an operator plays a 
simulation with virtual object(s) in a mixed real/ty space including a virtual space and real 
space comprising: 

a viewpoint detection program stdp for detecting the location/posture of a 
viewpoint of the operator; 

an inputting program step for inputting a real space image corresponding to the 
location/posture of a viewpoint of the operator; 

a geometric information acquisition program step for acquiring geometric 
information of real object(s); 

a r e c o gni t i o n program st e p for recognizing a curren t r e la t iv e rela t i o nship 
b et we e n t h e virtual o bj e c t (s) and r e al obj e ct(s); 

a rule memory for storing ifules for controlling the action of the virtual 

object(s); 

a computation program step for determining the next action of the virtual 
object(s) by referring to the rule memory based on a relation among the location/posture of 



a viewpoint of the operator, location(s) of the virtual obiect(s) and the geometric 



information of the real obiectfs) in acc or dance with t he rul e s s t o r ed in said rule m e mory 
and in correspondenc e with th e loeati o li/postur e o f t h e r e al obj e c t (s), and c o mpu t ing th e 
l o ca t ion/pos t ur e of th e virtual objee t (s)laft e r t he determin e d ac t i o n ; and 

a presentation program steplfor generating at least one image of the virtual 
object on the basis of the location/posture of the virtual object(s) after the action and the 
location/posture of the viewpoint positiomof the operator, and for representing the mixed 
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reality space to the operator by superimposing the image(s) of the virtual object(s) on the 
operator's view of the real space. 



